meaning in context

Even with the context you provided, and being a native speaker, that sentence tripped me up a bit too, mostly because it's employing a unfamiliar sense of the word guard (which would have been more familiar to readers at the time Scott wrote Waverly).

So I'll tell you both what the word means and how I puzzled it out, which might help you in similar situations in the future.

First, to answer your question directly, in this context, guards means:

the posture of defence or readiness in fencing, boxing, cricket, etc.

Collins sense 20

The hints that this is the sense being employed are that the kid is going through the guards with a weapon, and that weapon usually hangs over the chimney (in the selfsame study), and the nephew was surprised at the entrance of his uncle, all of which tell us it is likely the kid took the sword off the wall and was practicing fencing with it.

In particular, he was practicing the guards of that kind of weapon; guards here meaning the defensive positions in fencing, similar to the images below (though these guards are specific to the longsword, not the broadsword as in the story):

image credit: Pinterest

And, in case it's not clear to non-native speakers, through here is used in the sense of "from A to Z"; there is a fixed series of guards (as in the illustration above), and the kid was practicing each, one after another: he went through the series.

In short, the kid was play-fighting with the family's heirloom sword when his uncle walked in and nearly caught him.